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ABSTRACT 

Fifty instructors, from nine school districts and 
Five 4-year institutions, are taking part in the SET 

program — Students Evaluate Teaching. Research shows that (1) students 
can and do make fairly sound ratings of their teachers; (2) they tend 
to aaree with each other; (3) teachers rated best by students tend to 
obtain highest student gains; (4) student ratings often differ' from 
those by principals, supervisors, and other teachers. SET is an 
effective way for students to express their opinions, valid or 
otherwise. A visiting instructor is requested by any educator who 
feels the session can help him improve his teaching. The visitor 
leads a student discussion, during which he elicits their comments. 
The requesting teacher prepares his class for the visit, urging 
candid comment. Both teachers assure the students 
anonymous remarks will be used only for improving 
teaching methods. (This report includes a copy of 
discussion agenda.) Transcripts of the session, a 
the two instructors, and a class discussion after 
the evaluation. The experiment’s immediate objectives are to give a 
chance for the teacher to improve his instructions and the student to 
express himself on the programs. It is a way to define a group’s 
concept of what makes one member of an occupation more effective than 

application of the 
program are 



that their 
course content and 
the visitor’s 
conference between 
the visit make up 



another. The SET evaluations 
Critical Incident Technique, 
discussed. (HH) 



will be raw data for 
Future plans for the 



490 ^47 



. ^ U.S. DEPARTMENT OF HEALTH, EDUCATION & WELFARE 

‘ * * OFFICE OF EDUCATION 

THIS DOCUMENT HAS BEEN REPRODUCED EXACTLY AS RECEIVED FROM THE 

PERSON OR ORGANIZATION ORIGINATING IT. POINTS OF VIEW OR OPINIONS 

STATED DO NOT NECESSARILY REPRESENT OFFICIAL OFFICE OF EDUCATION 

POSITION OR POLICY. Students Evaluate Teaching 



OJ 

(T^ 



K\ 

LPi 

r<\ 





UJ 




There is good reason to believe that student unrest, especially at the 
college level, has been partly a result of what young people feel is poor 
instruction. Students are no longer satisfied with a voice at a lectern 
hiding behind a list of impressive degrees. They want involvement with their 
teachers; they need to be able to communicate more with the establishment. 
Students are more anxious and less tolerant of unexplained delays, unnecessary 
red tape and poor teaching. The tenuousness of today’s living brought on 
by the advent of nuclear weapons and the Vietnam war, has created a new 
breed of students who want to ask questions, get honest answers and test 
their own ideas and opinions. 

With the belief that student opinion is of vital importance and that 
students can help in the improvement of instruction, a program, SET 
(Students Evaluate Teaching) , has been developed in which students are called 
upon to evaluate the instruction they are receiving. Representatives from 
nine school districts and five colleges and universities involving 50 
instructors are already participating in the program. There is evidence 
that student ratings are accurate and important. A patient evaluates his 
physician; a client evaluates his attorney; a reader evaluates an author; 
but a teacher is evaluated by his superior, whereas, he should be evaluated 
by his students. SET proposed to elicit this evaluation from students in 
a dignified and meaningful way. 

It is stated in a special project prenared for and published by the 
joint Committee on Personnel Procedures by the California School Boards 
Association that "There is a growing body of research literature on the use 
of student’s rating of their teachers. This literature probably is more 
favorable and more consistent than any other in the area of teacher evaluation. 



Almost without exception the research reports indicate that pupils can and 
do make reasonably accurate ratings of teachers o Their ratings tend to agree 
with each other, and the teachers who are rated best by pupils tend to obtain 
the highest pupil gains <> Pupil ratings often do not agree with ratings by 
principals, supervisors, and other teachers."^ 

William A. McCall^ in an article on Measurement of Teacher Merit further 
states that "At last we find some professionally competent judges to teaching 
skill, namely, the teacher’s pupils, especially after they have been taught 
by the teacher for nearly a year« Out of the mouths of children come more 
accurate judgments of teachers than that rendered by their peers and 
superiors. . 

The fact that evidence proves that students are often accurate in what 
they say is a great step toward encouraging recognition of the importance 
of student opinion; however, whether students are accurate or not, what they 
think is still important. For example, the fact that a third grader thinks 
hi£ clASsroom is messy or that a graduate student thinks his professor shows 
favoritism, may not be relevant because learning may thrive in an untidy 
room, and a seminar may be more productive to all members if a few members 
and the professor are carrying on a stimulating dialogue. However, the 
significant point is that the students believe that these things are valid; 
therefore, it is important for the instructor to underst and w hat the students 
have to say, SET gives students the opportunity for dignified and effective 
expression and involvement. It is a legitimate participation by students 
in academic affairs. 



(1) Howsam, R. B. "Who’s A Good Teacher?" Joint Committee on Personnel 
Proceudres - page 33, 1960 

(2) McCall, W. A. "Measurement of Teacher Merit", Raliegh, North Carolina: 
State Superintendent of Public Instruction, 1952 
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This experimental program attempts to improve instruction at all levels 
through a system which would provide an alternative to the old method 
of sitting in by an administrator. This latter method creates an artificial 
situation - one which does not yield an accurate picture of day-to-day 
classroom teaching. When the administrator enters the classroom, the instructor 
either puts on a ’’good show" or freezes which gives an unusually poor impression. 
If the instructor asks the students to write or type anonymously an evaluation 
of the class and his teaching, the student is inhibited by the knowledge that 
the instructor will probably recognize his handwriting, typing or vocabulary. 

An evaluation based on a check-off blank used by itself is far too limited 
in its scope o 

The technique of the program, SET, involves gathering information 
through an instructor-student discussion period in which a visiting instructor, 
in the absence of the resident instructor, elicits comments from students 
on many aspects of the course. The plan is based on the fundamental concept 
that most students when properly stimulated, will comment fairly and surprisingly 
accurately if they are convinced that the instructor is genuinely interested 
in their opinions. These opinions can be a valuable source of information 
for self-evaluation by the instructor. 

The visits are requested by educators at the primary, secondary, 
college and university levels who feel that the sessions can help them 
improve their teaching. The visiting instructor is chosen from another 
district or college. He is someone who is unknown to the students as they 
are to him, and no introductions are made. Thus it is hoped that an impersonal 
relationship can be maintained. 

The resident instructor used the following proc dure in preparing his 
clcss for the evaluaticn sessiOr, which he reade aloud to the class before 




the next session of the class at which the visiting instructor appears. 



1 . 



I have arranged for this class to participate in a SET program. 

A visiting instructor who should be unknown to you will come to the 
next session of this class. He will lead you in a discussion 
evaluation of this class and cover such subjects as the content, 
outline, scope, teaching methods, text, and the instructor. I 
sincerely request that for my benefit so that I can imporve my 
instruction you discuss these areas openly, candidly, honestly, 
and constructively. Don’t feel that you need to (1) protect me 
in the discussion, (2) flatter me, or (3) tear me down completely. 
2. Remember — there will be no identification of anybody of anything 

said during this evaluation session; so again, I urgently 
request you to speak out freely at the next session of this class 
when you will be evaluating this learning situation. 

I 

When the visiting instructor meets with the class, he reads aloud to 
it the following statement; 

1. I am here at the request and on the invitation of your ifistructcr 

(name) . He is hopeful that during our discussion 

information will be revealed which will enable him to become a 
better instructor both through his own presentation and his 
organization of the course. We have with us today a secretary, 
who is unknown to you, who will take shorthand. It is important 
for you to understand that the transcription of your discussion 
will go only to the resident instructor. Furthermore, and 
equally important, there will be no identification of who makes 
which comment during today’s discussion. 

While presiding over our evaluation discussion today, I will follow 
an agenda that was designed to elicit germane information related 
to this course. In general, we will stay with the agenda, but 
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there will be room for Ideas that are of concern and may be outside 
the agenda. 

With this brief explanation to the class, the visiting instructor uses 

the following agenda and proceeds with the evaluation. 

Discussion Agenda 

I . Understandings which students have about the nature of the course 

1. What are the objectives in this course? 

2c What is it you're supposed to know or be able to do as a result 
of having taken the course’ 

3. Do the laboratory work and the on-campus work fit together? How? 

4, Who takes this course? (Juniors, seniors, graduates?) 

5o Is the course part of a sequence? Where does it fit in the sequence 

6. What is the relation of the course to other teacher preparation 

courses? 

7; If there is confusion about the purpose of the course, is it 
because students may be taking the course out of the intended 
sequence? 

^ ^ ■ Purpose . Role , and use of the learning materials 

1. What texts are used? 

2. How are they used? 

3. Do they fit the purposes of the coarse? 

4c What ki^iids of learning materials should be used? Why? 

Ill . The teacher ami his methods 

L, Does he know his subject’ Is he expert? 

? Doss ha come to class prepared? 

y wi'ia!! te;=iiching methods does Ke use? Are they appropriate to his 

purposes? 

How does he feel aboxjt hiS stviidents? 



5. Is he available to students both in class and out of class? 

6. If you seek him out, what do you seek him for? 

(Here the intent is to find out whether they ask only about 
"safe things," or about assignments, or for personal guidance, or 
for enjoyment of him as someone to chat with,) 

7. How do you decide what to write about? 

8. Are the topics relevant to what the class is about? 

9. Does the teacher critique your papers? 

10. Are his comments helpful? 

11. How are the papers graded? 

12. What else in the work is graded? 

13 o On what basis are gardes given? 

14. How is your final grade determined? 

15. Is the basis for the grade valid? Fair? 

16. What do you think ought to be garded? How? 

17. Is the weighting given to different kinds of work adequate? Fair 

18. Is there a better way to do it? 

19. What is the teacher’s general attitude toward students? 

20. Does the teaching have continuity? 

21c Does the teacher follow up or follow through ideas? 

22, Is the teacher enthusiastic about the teaching? The class? 

23, Does he have favorites? 

24, Do you think he’d like to do some things differently if he felt 
he could? 

25, Does the teacher stimulate you to do creative thinking? 

General Assessment 

■■■■ m I ■ ■ 

If you were to give grades, what grade would give the following: 

The teacher? 

The course? 

The texts? 



Transcripts of the sessions (which carry no names) are mads. These 
together with conferences between the two instructors, and a discussion with 
the class after the meeting by the resident instructor, make up the total 
evaluation. 

Nowhere in literature relative to instruction and the improvement and 
evaluation thereof has the SET technique been employed and except for the 
works of Flanagan^ Jensen^^ nothing in the literature in the opinion of 
the author relates to this area, and it relates only as techniques for 
handling the findings of the experiment. 

As is stated above, there are two main Immediate objectives in 
this experiment: (1) the opportunity for a teacher to improve his own 

instruction, and (2) the opportunity for students to express themselves 
regarding the kinds of instructional programs they are receiving. There 
are, however, long-term experimental possibilities that will hopefully get 
into the realm of evaluating objectively the instruction evaluations that 
have been procured in the manner described above. It is currently planned 
to use the Critical Incidence Technique for this objective evaluation, an 
invention of John C. Flanagan^ v^ho has used it in a number of situations. 
Travers^ describes this technique as a method of "observation, but it also 
involves the judgment of the observer Concerning what should be observed and 
recorded. The technique, is a method of defining the group’s concept 
of what makes a particular member of an occupation more effective than other 
members. It is a way of defining, for, example, what superintendents mean 



fl) Flanagan, John C,, Design for a Study of American Youth, Fc^ston: Houghton 
Mifflin, 1962, 

(2) Jensem, Alired "D^ermining Critical Requirement? of Teachers," Journal 
of Exp erim ental Educa tio Yi XX v 1951-52), 79-85. 

( 3 ) Trsvers, Robert M. W., An IntT oductlon to_ Educat i onal Res e ^ ch, Second 

Edition, New York: The MacMillan Company., 1964. 
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when they say that teacher X is more effective than teacher Yo” Travers 
goes on to say that It helps to avoid difficult situations that are created 
by superintendents and principals when they ask the question, "What makes a 
good teacher?"; and answers to these kinds of questions are non-specific 
and vague and give rise to other questions such as whether the teacher is 
considerate and kind and provides effective incentives, etc. Not all teachers 
may be described in such nice terms, and we do not all conform to the same 
criteria; so it is not going to be possible to measure development on the 
basis of these generalizations. Hence, Flanagan’s Critical Incidence 
Technique is supposed to be used to describe more adequately what a person 
has in mind when he says that one teacher is better than another o To refer 
to Travers again, o othis does not in any way mitigate the fact that the 
technique is still operating in the domain of judgments. In the ultimate 
analysis, the pronouncement by any person of what makes a good teacher must 
be based upon a reason judgment that this type of student learning is better 
than that." 

In 1951 Jensen^ used a Critical Incident Technique, and he described 
the critical incident as an observed teacher behavior or aspect of teacher 
behavior that is judged to make the difference between success and failure 
in teaching. The term "judged" indicates what is being accomplished, which 
is to be summarized in terms of behavior incidents. We must bear in mind 
that there is very little point in collecting descriptions of commonplace 
behavior that is about as typical those teachers judged to be good as 

of those judged to be poor. We must note that this method is not a scientific 
device for deciding what constitutes good teaching. It is only a method 



(3) Travers, Robert M,W., Aci Introduction to Educational Research , Second 
Edition, New York: The MacMillan Company, 1964 

C2) Jens'^r;- Alfred, "Determining Critical Reqi-ire nents of Teachers," Journal 



for describing what some personal group considers to be crucial matters 
in judging the merits of teaching, 

Jensen asked those cooperating in this stuc.y to first think of an 
ineffective teacher and then to relate an incident that made him say he 
was an ineffective teacher or irtade him think he was incompetent; and the 
same was done for effective teachers, Jensen also asked participants in 
his experiment to try to recall in their own childhood classroom incidents 
that were effective and ineffective on the part of their teachers. 

The raw data used in the Critical Incident Technique will consist 
essentially of the transcripts resulting from the SET classroom evaluation. 
Gathering this data together and making some meaningfulness out of it, is 
obviously a tremendous task and will require considerable work and research. 
Travers and others interested in the Critical Incident Technique have recog- 
nized it to be an extremely laborious one. It involves masses of data and 
techniques that are time consuming. 

In the coming months, a pool of visiting instructors will be trained 
as experts in eliciting responses in the classroom. It is hoped that the 
SET program will go on a four-year basis with a director and two assistants. 

The director and one assistant will have the task of channeling information 
about the program throughout the country, setting up evaluation situations 
and arranging for feedback of results. The second assistant will be involved 
with the accumulation and treatment of data. 

The organization of an advisory council is in process. The following 
professors have already accepted positions on this council: Dr. Kenneth Bailey, 

University of California, Irvine; Dr, Dan Dawson, University of Southern 
California; Mr. Preston Hatchkiss, Manager, Rixby Ranch, Los Angeles; 

Di . Hazel Jones, California State College, Fullerton; Dr, William Langsdorf, 
^resident, California State College, Fullerton; Dr Fillia.n Strand, Stanford 
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University; Mr. Leo Tamamian, Manager, Manufacturing, Hughes Aircraft, Fullerton; 
Dr.. Proctor Thompson, Claremont Men’s College; and Dr. Stanley Williams, 
California State College, Long Beach. A financial advisory committee headed 
b\ tk. Harold Jackson, Vice President, Bank of America, will assist in 
accu'lring and handling funds for the operation of the program. 

More students are serious today about their education than any time 
before in our history. They will accept a subordinate role with their 
instructors only in their comparative levels of knowledge. Above all, 
students want to relate honestly with their institutions. SET helps to 
pr^'.'vide this relatedness. 

It should be remembered that there are two keys to the success of 

SET: 

1, Students have the opportunity to speak candidly about their 
educational program. 

2. The students’ remarks are the confidential private property of 
the instructor about whom they are made. 




COMMENTS MADE M INSTRUCTORS AND STUDENTS 
WHO PARTICIPATED IN THE SET PROGRAM 



Student Comments 

I, Sixth gtader - "Sometimes she :.sn’t always here before us. We 
have to wait outside.- Sometimes she‘s in a bad mood. We were 
in here before her once and someone had left a big cigar on her 
desk and she was in a bad mood and she got real mad*" 

2„ Sixth grader - "She always says she will assign these essays, 
but no one has ever had one yet." 

3- Third grader - "He goes from sub 3 ect to subject and knows what 
he is going to do next." 

4. Fifth grader - "Tes, she likes everyone even if they don’t like 
her." 

5, Sixth grader - "He takes his contact lenses out in tront of 
the class •“ I squirm." 

6, Graduate student - "How many professors would let you speak out 
the way we. have in this class?" 

7. College student - "It’s a good program and it should be used for 
3,11 classes. One day a semester set aside for this wouldn t hurt. 



Instructors’ Comments 
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College - "When 1 entered the class, which consisted of college 
seniors, and I told them why I was there, their first, response 
was one of disbelief that anyone was Interested in their opinions. 
After I convinced them that. it. was ’for r.ea..y’ the discussion got 
underway in a mos^ relaxing, refreshing, and revealing way," 

Elementary - "I think the SET experience is very worthwhile. I 
not only gained from seeing the reports on my teaching, but, I 
found I was learning something about young people in general." 



Secondary - "Certainly this evaluative technique should be furthered 
It appears that it would be of great help to almost all teachers. 



4, Elementary - "In evaluating the students’ response, T feel sure 

there must be several areas where I might .improve in te.rms of teaching 
effectiveness « " 



College • "To summarize my feelings concerning the SET experiment, 
I believe it wa-5 a beneficial and worthwhile experience." 
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6 Elementary - “I think this tGchnlque, this program, made me stop 
and think more than anything else*” 

7c University - "At the college level I find that students have some 
pretty good ideas of their own about the course material » It helps 
to point up the question of what should be taught* Career-oriented 
students have some pretty good ideas about an answer to this question 
in professional courses*" 

8 Elementary - "SET is a concept as simple as a safety pin; it makes 
you wonder why nobody thought of it before because it works simply 
and so beautifully* Also, it is very significant to me the impor- 
tance placed on the individual self-improvement of the instructor*" 

9* Elementary - "They didn’t like my personal habit of grabbing them 
by the neck when I wanted to solve a problem." 

10* University - "As far as the college level is concerned, it doesn’t 
take too much to convince the students that everything is off the 
record and won’t be used against them or their professor*" 

11, College - "I feel the evaluation was a good joint effort between 

the students, faculty, and administration* All three are involved 
in this." 



Aider. W* Smith 
SET Program 
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